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Abstract 

The purpose of this study was to investigate effect of 
vocabulary comprehension in underprepared freshmen student 
responses on their rating of faculty teaching performances. 
Learning Skills Center students have weak vocabulary 
comprehension that might hinder their accurately rating the 
instructors. A vocabulary quiz was given to study skills 
students to determine their vocabulary comprehension of the 
evaluation form. It was found that less than half of the 
students understood 75% of the vocabulary words used in the 
evaluation form. Next, several sections of study skill classes 
were randomly divided into control and experimental groups. 
The word meanings were explained to the experimental group. 
A postt^ - ^t only, control group design was used in this study to 
determine the effect of comprehension on their rating of the 
instructors. The student evaluation of faculty teaching 
performance was then administered to all the students. The 
results of evaluation indicated no difference in rating between 
the control and experimental group for individual instructors 
at .05 level or in the composite experimental and control 
groups. 



ERIC 



3 



Vocabulary Comprehension 

3 

Comprehension of Vocabulary of Evaluation Form: 
Its Influence On Student Rating of Faculty 

Student evaluation of the faculty teaching performance is 
done routinely in most colleges and universities. This 
evaluation is done to measure the teaching effectiveness of 
instructors. These evaluations have an important effect on the 
decisions of promotion, merit pay, and tenure of the faculty 
members. At Delaware State University (DSU), a lengthy 
evaluation form is used by all students to evaluate faculty 
performance. 

It has been a faculty perception that the underprepared, 
academically unmotivated students do not give an accurate 
rating to their instructors because of weak vocabulary 
comprehension. Underprepared students are often frustrated 
by words which they do not understand and will either 
misinterpret the question or give any answer just to respond 
and get on with the task at hand. Rather than admit their 
ignorance in front of others they will pretend that they know 
the meaning. 

The underprepared freshmen students who are enrolled in 
the reading and study skills classes at the Learning Skills 
Center have weak vocabulary comprehension. It is likely that 
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they do not fully understand the meaning of the many words 
used in the evaluation form. If less than 80% of the students 
understand only 75% of the vocabulary words used in the 
evaluation form, then explanation of such words prior to their 
evaluation of the instructor should have an influence on their 
rating of the instructor. 

At DSD, the Office of Testing developed an evaluation 
instrument, using several other nationally known evaluation 
instruments as models. This instrument of student evaluation 
of instructor and course has been in use for at least four years. 
The evaluation form is used by a broad spectrum of students at 
the Delaware State College, from students at the Learning 
Skills Center who are enrolled to strengthen their reading and 
study skills, to graduate students in various academic 
disciplines. 

Academically underprepared are those students whose 
scores fall below the local median on Sequential Test of 
Educational Progress (STEP), (Miller, DSU Policy, 1976). The 
students are placed in the Learning Skills Center Program 
because of their low entrance test scores on STEP. They must 
satisfactorily complete courses in areas of Reading, Study 
Skills and Writing in the first three semesters at the College. 
Students earn a grade of Satisfactory or Unsatisfactory rather 
than a letter grade. Their comprehension of vocabulary is 
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usually weak. Comprehension is defined (Webster,! 988) as an 
act of comprehending or understanding. 

The evaluation instrument is composed of 58 questions; 
it evaluates the general lecture/discussion period within a 
classroom. On the reverse side of the instrunrient there are \B 
additional questions that evaluate the accompanying 
laboratory courses. There is also a blank sheet for students to 
write their comments. Along with this form, an answer sheet 
which can be read by a computer is provided. The evaluation 
ratings are nunfierical and range from 5.0 (the highest) to 1.0 
(the lowest). 

The Learning Skills Center students who are deficient in 
reading comprehension may give a rating without 
understanding the meaning of words used in the questionnaire. 
There are many words such as irrelevant , stressed, adequately , 
and promptly , used in a sentence format in the evaluation 
instrument. A misunderstanding of the meaning of these words 
can result in an inaccurate rating of the instructor. As 
freshmen, students might not value the importance of the 
evaluation and how it may affect the instructor in his or her 
professional career. 

Instructors have often felt that the kinds of questions 
that are asked require a discerning mind of a conscientious 
student who truly understands the questions. An instructor's 
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advancement hinges on these evaluations and to receive a fair 
evaluation is critical. This study is considered important 
because students' ratings of instructor are not always 
accurate, yet promotion, merit pay and tenure are often based 
on the results of these evaluations. 

In an extensive search of the literature, it became 
apparent that research conducted on underprepared students 
and how their vocabulary comprehension influences their 
rating of instructors has been limited. What follows are 
examples of what currently appear in the literature. 

Whittington (1984) reported that student evaluations of 
faculty began in an era of accountability, and the process was 
ritualized even though the campus atmosphere was not 
rebellious in the mid eighties. Belcher (1991) reported that 
students gave higher ratings to their instructors if they 
believed that they were getting good grades. The students who 
did not feel that they were doing as well in class gave less 
favorable evaluations. Marsh (1987) found that student ratings 
were primarily a function of the instructor rather than of 
course content. Feldman (1987) stated that teachers got 
higher ratings for their work with senior classes than when 
t!iey worked with the lower levels. Also, the teachers 
received lower rating for classes which were compulsory than 
for those which were elective. 
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Olagunju and Jordan (1982) found that academically 
underprepared and unmotivated students had more complaints 
about learning the course materials than the academically 
motivated students. They also found that the overall rating 
score of the instructors was considerably less than the rating 
score obtained from highly motivated students. 

The literature contains inconsistent data regarding 
evaluations. Jacobs (1987) found that students take faculty 
ratings seriously, although this finding conflicts with the 
perceptions of faculty. McKeachie (1979) stated that student 
ratings should not be used as the single measure of teaching. 
If student ratings are used for personnel decisions, then 
instructors should have a say in the interpretation of the data. 

Miller (1988) made a recommendation that student 
evaluations when used in support of decisions for tenure or 
promotion, be evaluated in the light of extraneous features of 
the courses (such as compulsory or optional, lower level or 
senior level). Raskin and Plante (1979) observed that student 
ratings place pressures on instructors that few instructors 
will choose to ignore in days of budgetary constraints and 
retrenchment. 

Machina (1 987) noted that the student ratings should not 
be the sole criteria used in personnel decisions. When 
students underrate a faculty member, it is an indicator of 
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break in the teaching process. Machlup (1979) supported 
regular evaluations of teacher performance, but cautioned 
about the high ratings of the teacher as justification of 
decisions for promotion, tenure and merit pay. 

Wasfi (1989) considered the student evaluation to be the 
only objective criterion for gauging teaching effectiveness, 
although he felt a short questbnnaire (about 20 questions) 
would suffice. Aleamoni (1987) surmised from his extensive 
research background on student evaluation of faculty that 
many faculty concerns regarding the student's immaturity or 
capriciousness in judging teaching effectiveness were 
unfounded. 

Harris (1982) has stated that experts' consensus is that 
student rating instruments for faculty evaluation have been 
refined and modified to the extent that their validity and 
reliability is no longer questionable in summative evaluation. 
She discovered that when responsibility of constructing the 
instruments was left up to individual departments, validity, 
reliability and consistency throughout the institution were 
found to be jeopardized and results inconsistent. 

Student evaluations can play a significant role in the 
advancement of instructor's professional career. Teaching 
performance evaluation is extremely important to the 
instructor. It would be an interesting study to see how 
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infiportant the freshmen underprepared students with weak 
vocabulary comprehension perceive this process to be. 

The primary purpose of this study was to add empirical 
data to the body of existing literature on faculty evaluation. It 
was the intent of this study to explore the effect of 
vocabulary comprehension on faculty evaluation. If vocabulary 
comprehension of the evaluation questionnaire is a factor in 
rating of instructors then increased comprehension should be 
of benefit to the instructors in the form of more favorab's 
ratings. 

The vocabulary of the evaluation questionnaire contains 
many words which students might not fully understand. If that 
is indeed the case, it will become evident from the results of 
the vocabulary quiz, which all students will take. The 
vocabulary quiz will contain words from the evaluation 
questionnaire and will be in a sentence format similar to the 
questionnaire. 

It was anticipated that there will be a difference in the 
rating of the instructors by students who are taught the 
meaning of words prior to their completion of the evaluation 
instrument. Their rating should show a difference from the 
rating of those students who did not receive any instruction. 

The university from which the data was obtained is a 
progressive liberal arts institution serving a diverse student 
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population. The students participating in this study were 
derived from the study skills classes of the fall of 1992. 
Students of three different professors participated in this 
study. The three professors each have a teaching experience of 
twenty years. Each section of a study skills class was 
randomly divided to identify an experimental group and a 
control group. 

The experimental variable was the teaching of the word 
meanings used in the evaluation questionnaire. The outcome of 
this treatment should yield a more accurate rating of the 
instructor since the criteria are better understood. 

The study was necessarily limited to the fall 1992 
freshman class entering DSU, Dover, DE. This study was also 
limited to the extent that the sample of academically 
underprepared students was primarily African-American, 
which might not be representative of the total population. 
This study might be limited due to the fact that student's 
consent and prior knowledge of their participation in this 
experimental study were institutionally required 

This study delimited vocabulary comprehension to the 
words or phrases used in the evaluation questionnaire. 

Method 
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Campbell and Stanley's (1963) design #6 was used for 
this study. In this design, no pretest is given, only a posttest 
Is used for both control and experimental groups. This design 
is suitable for situations where a pretest would be awkward 
and the stuflent anonymity must be retained. This design is 
strong in both internal and external validity. Extraneous 
variables such as history, maturation, testing, 
instrumentation, regression, selection, and interaction of 
selection and maturation are controlled and are not threats to 
the internal validity. The external variables such as 
interaction effect of testing was controlled because this 
design does not require a pretest. 

Control and experimental groups were randomly selected 
from each section of the study skills classes. An 
experimenter made quiz using vocabulary words contained in 
the questionnaire and designed in a sentence format was used. 
The individual professors administered and graded the 
quizzes using agreed upon criteria for correct responses. 
Subjects 

Students enrolled in the study skill classes taught by 
three different professors participated in this study. It was a 
predominantly African-American group. Many of these 
students were non traditional students. The students were 
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apprised of the nature of the study and their participation was 

voluntary. 

Procedure 

In order to test the vocabulary comprehension of the 
students of different professors who teach two sections each 
of study skills courses, a vocabulary comprehension quiz was 
given. This quiz contained 20 words chosen from the 
evaluation questionnaire and used in a sentence format. 

All the students were asked to respond as to whether or 
not they understood the meaning of words or not by placing a 
check mark on the two columns, as well as their personal 
definition of the word meaning. In this way the percentage of 
students who did not understand the word meanings in the 
context were identified. If 80% of the students did not 
understand 75% of the word meanings then the students will 
be judged to have difficulty with the meaning of words. If 
they did understand the meaning, then other factors had an 
influence. A comprehension level of 75% was chosen because 
that is the level required to obtain Satisfactory "S" grade in 
the LSC classes for successful completion of the course work. 
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VOCABULARY QUIZ 
Check the meaning of the underlined words in the sentences 
below in the right hand column (known or not known). Write the 
meaning of the underlined words in the personal defmition column. 



Meaning Not Personal 
Known Known Definition 

1. Does your class use su pplementary 

reading? 

2. Is the darity of material an aid to 

understanding? 

3. Are the course objectives maintained in 

the classroom? 

4. Is the room adequately equipped with 

light? 

5. Are your papers graded promptly ? 

6. Do you receive constructive feedback? 

7. Do you understand the criticism of 

your returned paper? 

8. Do you understand the objective of 

each class exercise? 

9. Is there any real life application of 

what you have learned in the class? 

10. Do you copy down the homework 

assignment accurately? 

11. Are you receptive to different styles 

of teaching? 
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12. Do you read books that are relevant 

. to your future career? 

13. Do you incorporate learning from 

different sources ? 

14. Do you understand the concepts of the 

subject taught? 

15. Do you know the principles of the 

subject matter? 

16. Do you find some matter irrelevant in 

discussion? 

17. Was the transition from one topic to 

another clear? 

18. Are the important points sgessed for 

your understanding? 

19. Is the subject matter taught creatively ? 

20. Do you review the material periodically ? 

Next, the students were randomly divided into two 
groups, the experimental group of students was taught the 
meaning of the words (Appendix B), while the other group did 
not receive such instruction. The control group was kept out 
of hearing range in another room while engaged in their daily 
activities. Immediately afterwards, the two groups were 
brought together and asked to complete the teaching 
performance evaluation of the instructor. Their answer sheets 
were color coded to keep the experimental and control groups' 
responses separate. Two separate class periods of 50 
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minutes were devoted to these activities by the classes that 
met on Monday, Wednesday and Friday schedules. For the 
classes that met Tuesday and Thursday for 75 minutes, a 
similar time period was utilized. 

The author wanted to see if the vocabulary 
comprehension affects the student ratings when the control 
group and experimental group's ratings were compared. 
Evaluation ratings were tabulated and statistical analysis was 
done on a personal computer, utilizing "Excel" program. An 
initial descriptive statistical analysis was done to determine 
if further inferential statistical analysis of data was 
necessary. 

It was hoped that this study would give valuable data 
concerning the vocabulary comprehension factor that might 
affect the student rating of the teaching performance of their 
instructor. 

Results 

To maintain the anonymity of the professors, they were 
designated letters A, B and C for the results and ensuing 
discussion. Results from the vocabulary quiz showed that 53% 
of the student (N=85) failed the vocabulary comprehension quiz 
at 75% level. In other words less than 80% of the students 
understood 75% of the vocabulary words used in the evaluation 
form. The data is presented in Table 1 . 
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Table 1 . 

Vocabulary Comprehension of Evaluation Form 



Professor/ 
Section 


No. of 
students 
taking test 


No. of 
students 
scoring <75% 


A1 


14 


11 


A2 


11 


6 


B1 


12 


8 


82 


17 


8 


CI 


18 


8 


C2 


13 


4 


Total 


85 


45 



An initial descriptive statistical analysis was done on 
the student ratings received by each individual professor. It 
was found that the ratings of the control and experimental 
groups did not differ significantly at .05 level. The analysis is 
presented in Table 2. 
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Table 2. 

Comparison of ratings of individual professors 





Professor A 


Professor B 


Professor C 




Cont. 


Expt. 


Cont. 


Expt. 


Cont. 


Expt. 


Mean 


4.09 


4.23 


3.58 


3.86 


4.20 


4.17 


Median 


4.12 


4.24 


3.67 


3.89 


4.18 


4.19 


Mode 


3.94 


4.18 


3.69 


3.94 


4.18 


4.19 


Std.Dev. 


0.34 


0.30 


0.29 


0.27 


0.23 


0.17 


Variance 


0.18 


0.09 


0.08 


0.07 


0.05 


0.03 


Kurtosis 


0.13 


-0.14 


-0.52 


-0.28 


-0.46 


-1.128 


Skewness 


-0.33 


-0.13 


-0.54 


-0.36 


0.22 


-0.34 


Sum 


208.5 


215.8 


182.4 


197.1 


214.1 


212.5 


Conf .level 


0.094 


0.081 


0.079 


0.073 


0.062 


0.046 



Note. Std. Dev. = Standard Deviation, Conf. level = Confidence 



level, Expt. = Experimental group, Cont.= Control group. 

Next, all the experimental groups ratings of the three 
different professors were combined and a composite of control 
and experimental group were compared. An initial descriptive 
analysis again failed to show significant difference at .05 
level. Table 3 shows composite ratings. 
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Table 3. 

Comparison of control and experimental ratings. 





Control 
Rating 


Expt. 
Rating 


Mean 


3.95 


4.09 


Std. Deviation 


0.40 


0.29 


Kurtosis 


-0.20 


0.38 


Skewness 


-0.38 


-0.29 


Conf. level (95%) 


0.06 


0.05 


Table 4. 






t-test of significance 








Control 


Expt. 


Mean 


3.95 


4.09 


Variance 


0.1 6 


0.09 


Observations 


153 


1 53 


Pearson Correlation 


0.75 




Pooled Variance 


3.50 




df 


280.80 




t 


-3.33 




P(T<=t) one-tail 


0.00 




t Critical one-tail 


1.65 




P(T<=t) two-tail 


0.00 




t Critical two-tail 


1.97 
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Table 4 shows the t-test of significance between two 
samples assuming unequal variances. 

Discussion 

As stated earlier in the report, this study was 
undertaken to determine the effect of vocabulary 
comprehension on faculty ratings by academically 
underprepared college freshmen. 

Findings indicated that there was no significant 
difference between the mean scores of the control and 
experimental groups at p>.05 level. Although the gains in the 
evaluation ratings of individual instructors A and B were small 
and were not confirmed to be statistically significant by 
analysis, they nonetheless looked better than those given by 
control group. These two professors also had larger number of 
students scoring <75% on vocabulary quiz, whereas professor C 
who had the least number of students scoring <75%, showed a 
slight decline in rating given by her students. 

Recommendations for further research include selection 
of a larger sample size, as well as tesiing another sample of 
ethnically balanced mixture of academically underprepared 
students from another four-year institution. Such a group 
might be more representative of the total population and 
provide data more suitable to generalization. Another 



20 



Vocabulary Comprehension 

20 

variation of the study might be to test the vocabulary 
comprehension after a two week period to determine retention 
or understanding of the words used. 

Although there was no statistically significant 
difference detected in the evaluation ratings, it is 
recommended that LSC students be explained the meanings of 
the evaluation questionnaire words prior to its administration. 
Invariably, the comfort level experienced by student is raised 
when confident or cognizant of the word meanings whether it 
be a test or evaluation or a survey, thus ensuring the accuracy 
with which the task is done. 
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